Augmented reality display system and method of display

ABSTRACT

The present invention describes a display system that includes a display, including a display screen; a viewpoint assessment component to determine a viewpoint of a user positioned in front the display screen; and an object tracking component to track the user manipulation of an object positioned behind the display screen.

BACKGROUND

Sensing technologies are available which make it possible to manipulatecomputer generated content via simple gestures in front of a displayscreen. Also, display technologies are available that allow a user toview the computer generated content in 3 dimensions (3D). Currentdisplay systems with gesture based control systems are often problematicin that the user's hands tend to block the user's view of the contentbeing manipulated on the display screen.

BRIEF DESCRIPTION OF DRAWINGS

The figures depict implementations/embodiments of the invention and notthe invention itself. Some embodiments are described, by way of example,with respect to the following Figures.

FIG. 1 illustrates a block diagram of a front view of a display screenin an augmented reality display system according to an embodiment of theinvention;

FIG. 2A shows a front perspective view of a desktop version of anaugmented reality display system according to an embodiment of theinvention;

FIG. 2B shows a side view of a desktop version of an augmented realitydisplay system according to an embodiment of the invention;

FIG. 2C shows a perspective back view of a desktop version of anaugmented reality display system according to an embodiment of theinvention;

FIG. 3 shows a view of the points required to generate the user'sviewpoint according to one embodiment of the invention;

FIG. 4A shows a flow diagram for a method of displaying an image foraugmenting an object behind a transparent display screen according to anembodiment of the invention;

FIG. 4B shows a flow diagram for a method of generating content fordisplay on a display screen according to an embodiment of the invention;

FIG. 5A shows a side perspective view of a desktop version of anaugmented display system where the user is manipulating a physicalobject behind the display screen according to one embodiment of theinvention;

FIG. 5B shows a front perspective view of the display system shown inFIG. 5A where the user is manipulating a physical object behind thedisplay screen according to one embodiment of the invention;

FIG. 6A shows a side perspective view of a desktop version of anaugmented display system where the user is manipulating a virtual objectbehind the display screen according to one embodiment of the invention;

FIG. 6B shows a front perspective view of the augmented display systemshown in FIG. 6A where the user is manipulating a virtual object behindthe display screen according to one embodiment of the invention;

FIG. 7A shows a perspective front view of a handheld version of anaugmented reality display system according to an embodiment of theinvention;

FIG. 7B shows a flow diagram for a method of generating content fordisplay on the opaque display screen shown in FIG. 7A according to anembodiment of the invention;

FIG. 8 shows a computer system for implementing the methods shown inFIG. 4A-4B and 7B and described in accordance with embodiments of thepresent invention.

The drawings referred to in this Brief Description should not beunderstood as being drawn to scale unless specifically noted.

DETAILED DESCRIPTION OF EMBODIMENTS

For simplicity and illustrative purposes, the principles of theembodiments are described by referring mainly to examples thereof. Inthe following description, numerous specific details are set forth inorder to provide a thorough understanding of the embodiments. It will beapparent, however, to one of ordinary skill in the art, that theembodiments may be practiced without limitation to these specificdetails. Also, different embodiments may be used together. In someinstances, well known methods and structures have not been described indetail so as not to unnecessarily obscure the description of theembodiments.

In one embodiment of the present invention, the display screen iscomprised of a transparent material so that a user positioned in frontof the display screen can easily see through it. Viewpoint assessmentsensors are positioned to capture information about a user and hisposition in front of the display screen. In addition the display systemincludes object tracking sensors positioned to track a physical objectpositioned behind the screen and manipulated by the user. The objectbehind the screen being manipulated by the user can be the user's ownhand or a physical object being held or otherwise manipulated by theuser. Responsive to data output from the viewpoint assessment sensorsand the object tracking sensors, an image is generated on the displayscreen that overlays or augments objects placed behind the screen. Theseoverlaid images are registered with the objects positioned behind thescreen based on the user's viewpoint (eye position). The sensed positionof the user's hands relative to the objects can be used to enablemanipulation of the overlaid images on the display screen.

Referring to FIG. 1 shows a block diagram of a front view of a displayscreen in an augmented reality display system according to an embodimentof the invention. The present invention is a display system 100comprising: a display 100, including a display screen 112; a viewpointassessment component 116 to determine a viewpoint of a user positionedin front the display screen 112; and an object tracking component 124capable of tracking the user manipulation of an object 120 positionedbehind the display screen 112. In addition, the display system also caninclude a display generation component 126, wherein based on data 128from the viewpoint assessment component 116 and data 130 from the objecttracking component 124, the display generation component 126 createscontent for the display on the display screen 112. The displaycontroller component 130 outputs data 134 from at least the displaygeneration component 126 to the display screen 112.

FIG. 2A shows a front perspective view of a desktop version of anaugmented reality display system according to an embodiment of theinvention. FIG. 2B shows a side view of a desktop version in FIG. 2A. Inthe embodiment shown in FIGS. 1-3 and 5-6, the display 110 includes adisplay screen 112 that is comprised of a transparent screen material.Viewpoint assessment sensors 140 a, 140 b are positioned to face towardsthe user to capture the user's head position or facial detail. In theembodiment shown in FIG. 2B, the dotted lines show the capture range ofthe sensors (both viewpoint assessment and object tracking). The capturerange depends upon the type of sensor used and the position of thesensor. Referring to FIG. 1, the viewpoint assessment sensor data 144 a,144 b is used by the viewpoint assessment component 116 to determine theuser's viewpoint.

In addition, the display system 100 includes one or more object trackingsensors 148 a, 148 b covering the space behind the display screen tosense objects (including the user's hands) positioned behind the displayscreen. FIG. 2C shows a perspective back view of a desktop version of anaugmented reality display system according to an embodiment of theinvention where the object tracking sensors 148 a, 148 b can be moreclearly seen.

Referring to FIG. 1, data 150 a, 150 b collected from the objecttracking sensors 148 a, 148 b can be used by the object trackingcomponent to determine the position (and, optionally, shape andorientation) of an object 120 behind the screen. Data (128, 130) outputfrom the viewpoint assessment component 116 and the object trackingcomponent 124 is used by the display generation component to generate animage on the display screen that can overlay or augment objects behindthe screen.

The display system includes a display screen 112 for displaying content.In one embodiment, the display screen is a transparent screen whichallows a user interacting with the display to see objects positionedbehind the display. In an alternative embodiment, the display screen isopaque but transparency is simulated by displaying a view of objectsbehind the display screen captured by a video camera that captures videobehind the display screen. In another alternative embodiment, thedisplay screen is capable of operating either in a first mode (atransparent screen) and a second mode (an opaque screen), and switchingbetween the two modes based on system or user controlled instructions.

Different screen materials may be used to provide the transparentdisplay screen 112. In one embodiment, the transparent display screen isa transparent OLED (organic light-emitting diode) screen. In analternative embodiment, the display screen is comprised of transparentLCDs (liquid crystal display). However, the transparent LCD screenimplementation assumes a well lit working environment. In a thirdalternative embodiment, the display screen is comprised of a partiallydiffusing material and the content is projected onto the partiallydiffusing material. Although many alternative implementations arepossible, the transparent display screen operates so that objectspositioned behind the display screen can be easily seen or viewed by auser positioned in front of the display screen. The transparent displayallows the user to have a clear view of the objects behind the screenthat are being manipulated in real time and to instantaneously see theeffect of their manipulation on the object.

As previously described, in one embodiment the display screen is capableof switching between display modes. For example, the display screenwould be transparent in one mode and in a second, opaque, mode thedisplay screen would operate as a conventional display screen. A displayscreen capable of switching between two modes could be implemented inone embodiment as a transparent OLED screen with a liquid crystalshutter positioned behind the OLED screen. In a first mode thetransparent LED screen would be transparent (liquid crystal shutteropen) and in a second mode the screen would appear to be opaque (liquidcrystal shutter behind the display screen closed) for use as aconventional display.

In one embodiment, the display screen 112 includes a frame 154 or bezel,typically located around the periphery of the display screen. Referringto FIG. 1, the display has both a front surface 156 and a back surface158 (shown more clearly for example in FIGS. 2 and FIG. 3). For theembodiment in FIG. 1, sensors that collect data for the viewpointassessment component, viewpoint assessment sensors 140 a, 140 b, arepositioned so that they are capable of determining the viewpoint of auser positioned in front of the display screen. Referring to FIG. 2Bshows dotted lines emanating from the viewpoint assessment sensor 140 a,the dotted lines indicating the approximate capture range of theviewpoint assessment sensors. In the example shown in FIG. 2B, the imageviewpoint assessment sensors are image capture devices intended tocapture the facial detail of the user 142.

Because in the embodiment described with respect to FIG. 1, the displayscreen is transparent, it is possible for the viewpoint assessmentsensors to be located in a position behind the display screen as long asthis sensor position provides a clear viewing angle of the user.However, in the example shown in FIG. 1, the viewpoint assessmentsensors 140 a, 140 b are positioned on the front side of the displayscreen so that content on the display screen, and its surrounding bezel,does not interfere with the sensed information.

Referring to FIG. 1 the perimeter of the object tracking sensors 148 a,148 b on the back of the display are represented by dotted lines.Sensors that collect data for the object tracking component, objecttracking sensors 148 a, 148 b, are positioned so that they are capableof capturing data about the objects (including the user's hand)positioned behind the display screen. Referring to FIG. 2B shows dottedlines emanating from the object tracking sensor 148 a, the dotted linesindicating the approximate capture range of the object tracking sensor.As previously described with respect to the viewpoint assessmentsensors, because the display screen can be transparent, it is possiblefor the object tracking sensors to be located in a position in front ofthe display screen. However, typically the object tracking sensors arepositioned on the back side of the display screen so that content on thedisplay screen, and its surrounding bezel, does not interfere with thesensed information.

Although multiple viewpoint assessment and object tracking sensors areshown around the boundary of the display screen in the embodiment shownin FIG. 1, it is possible for the viewpoint assessment or objecttracking to be implemented using a single sensor. For example, thesensors 140 a and 140 b could be replaced by a single sensor 140 thathas a wider coverage range so that it essentially covers the same rangeas sensors 140 a and 140 b combined. Since the coverage of the sensorsis determined by the capture range (both distance and angular coverage)of the sensor or the combination of sensors, the sensor or combinationof sensors should cover the anticipated position of the user (in thecase of the viewpoint assessment sensors for the embodiment in FIG. 1)or the anticipated position of the tracked object (in the case of theobject tracking sensor for the embodiment shown in FIG. 1).

Although the term “user” is used throughout, when used with respect tothe viewpoint assessment sensors, the term user refers to the featuresof the user (eyes, head and shoulders, face, etc.) necessary for theviewpoint assessment component, to determine the user viewpoint. Withreference to the object tracking device, the object being tracked is aphysical object capable of being manipulated by a user. The trackedobject being manipulated by the user behind the display screen, can bethe user's own hand or alternatively, another physical object (e.g.block, pliers, etc.) that the user is directly or indirectlymanipulating. For example, a user could be directly manipulating aphysical object (the block shown in FIGS. 5A and 5B) or indirectlymanipulating a physical object (for example (not shown), using his handto hold pliers which he uses to manipulate the block).

The display screen can be the size of a large desktop monitor, as shownin the FIG. 1, or the display screen can be smaller—such as a notebookcomputer display screen or mobile device screen. In the case of a mobileor handheld device, the user would typically hold the device with onehand and look at objects or scenes behind the mobile device by lookingthrough the mobile device screen.

Referring to FIG. 3 shows a view of the points used in oneimplementation to generate a viewpoint from the perspective of a user.As shown in FIG. 3, the location of the user's viewpoint in front of thedisplay screen 112 is designated by the symbol (V), the location of theobject located behind the screen is designated with the symbol (O). Thelocation of the content on the display screen 112 is designated by thesymbol (P). According to one embodiment of the invention, content isgenerated based on the user's viewpoint and the location of the trackedobject.

FIG. 4A shows a flow diagram for a method 400 of displaying an image foraugmenting an object behind a transparent display screen according to anembodiment of the invention. The method shown in FIG. 4A includes thesteps of: determining a user viewpoint based on data collected by theviewpoint assessment sensors (step 410); determining the location (and,optionally, shape and orientation) of an object capable of beingmanipulated by a user behind a display screen, wherein the objectlocation is based on data collected from at least one object trackingsensor (step 420); and generating content for display on a displayscreen, wherein the generated content location is based on the viewpointwith respect to the location of the object positioned behind the displayscreen (step 430).

Although different definitions of user viewpoint can be used, withrespect to FIG. 1 the location of the user viewpoint is described (forpurposes of discussion only) as the position (V) located midway betweenthe user's two eyes. Eye position or location can be found using varioustechniques including through the user of face detection software or byapproximation from the position of a tracking device being worn by theuser.

In one embodiment, determining the user's eye position is found using IR(infrared) illumination. In this implementation, an IR illuminationsource floods IR light in the vicinity of the user. In this case, theviewpoint assessment sensors 140 a, 140 b are IR sensitive devices,typically an IR sensitive camera. IR light is reflected off the user'sretina and the IR sensitive camera (the viewpoint assessment sensor)captures the user's eye location relative to the display screen and theobject positioned behind the display screen. Based on the location ofthe user's eyes, the viewpoint assessment component determines theuser's viewpoint with respect to the screen and the object.

In an alternative embodiment, information about the user's eye position(used to determine the user's viewpoint) can be achieved using facedetection software 180. In this case, the viewpoint assessment sensors140 a, 140 b can be simple cameras. Based on the feature detectionsoftware 180 in the viewpoint assessment component 116, the eye positionof the user is located. Based on the eye position of the user, the userviewpoint can be determined.

In an alternative embodiment, some form of active tracking of the user'sposition is used. For example, the user could wear a device that iscapable of being tracked in 3D space. In one implementation,retro-reflective markers could be applied to an object worn by the user(i.e. a headset, a pair of glasses, etc.). Based on the position of thetracked object, the viewpoint assessment component software 116 providesan accurate approximation of the user's eye location.

In another alternative embodiment, the eye location of the user can beapproximated using depth cameras as the viewpoint assessment sensor 140a, 140 b. In this implementation, a depth camera is used to locate theuser (user's head and shoulder at a depth or depth range nearer to thedepth camera) and the background (at a depth or depth range further fromthe camera). The viewpoint location can then be approximated as being atthe center of the head shaped region in the depth image. Any of theabove techniques listed in the preceding paragraphs can be combined togive a more accurate determination of the viewpoint location.

In one embodiment, the display is an auto-stereo display. In thesedisplays, after determining the location of each eye, different viewsare provided for each eye. The display system thus needs to know thelocation of each eye, so that content can be generated in the rightlocation for each eye. Some of the embodiments described above give theposition for each eye directly, in those that don't the typical humaninter-occular distance can be used to calculate eye positions from asingle viewpoint.

Referring to FIG. 2C shows a block diagram of a back perspective view ofa display screen in an augmented reality display system according to anembodiment of the invention. In the embodiment shown in FIG. 2C, theobject tracking sensors tracks the physical objects (e.g., user's hand120 a, and physical cube 120 b) physically positioned behind the displayscreen. In the implementation shown in FIG. 2C, object tracking sensors148 a, 148 b are shown located or embedded in the frame 154 locatedaround the periphery of the display screen, collect information about anobject positioned behind the back surface 158 of the display screen 112.

Although the object being manipulated by the user can be a physical orvirtual object (an augmented image) being displayed on the displayscreen, the object being tracked behind the display screen by the objecttracking sensors is a physical object. Further, a single or multipleobjects may be tracked. For example, in the case of a user manipulatinga virtual image as shown in FIGS. 6A and 6B, the physical object 120 alocated behind the display screen and being tracked is the user's hand.In the example shown in FIGS. 5A and 5B where the user is manipulating arectangular block behind the screen, there are two physical objectsbehind the display screen being tracked—the user's hand 120 a and therectangular block 120 b. The user can manipulate the block so that thedisplayed object (the car) appears to be inside of the block..

Although many types of object tracking sensors 148 a, 148 b can be used,in one embodiment the object tracking sensors are image capture devices.In one implementation, the image capture device (object tracking sensor)is a depth camera. For applications that require sensing gestures andobjects in the volume of space behind the screen, a depth camera wouldbe appropriate as it provides information regarding the shape andlocation of the object in 3D. For example, for the case where thetracked object is the user's hand, the depth camera would provideinformation regarding where the user's fingers are and the position ofthe user's fingers in relationship to the second tracked object (therectangular cube.)

Although depth cameras may be used to provide information about theshape and location of the object in 3D space, other types of sensors andtechniques for determining position and shape of the tracked objectcould be used. For example, sensors and techniques that are used forsensing gestures and object in front of the display screen, might alsobe applied for tracking objects behind the display screen.

In one embodiment, the objects to be tracked have a code embedded on theobject or registration marks on the object and the object trackingsensor is a camera. For example, a barcode label could be attached tothe tracked object 120. In this case, the camera capturing an image ofthe code or registration marks and based on the location of those marks,could also determine where the object is physically located. Further,software can interpret the code or markings to provide information aboutthe object (i.e. size, shape, use instructions, etc.).

The display controller component 130 includes: a viewpoint assessmentcomponent 116, an object tracking component 124 and a display generationcomponent 126. The display generation component 126 creates an overlaidimage that is displayed on the transparent screen that provides anatural way to augment user interactions with real world objects(located behind the display screen). Because the display screen 112 istransparent, the user can easily view and interact with both thephysical object 120 located behind the screen and the overlaid image 510displayed on the transparent display.

The display system creates an “overlaid” image 510 on the display screen112—where the overlaid image 510 is an image generated on the displayscreen that is between the user's viewpoint and the object 120 behindthe screen that it is “overlaid” on. The overlaid image is dependentupon the user's viewpoint. Thus, the position of the overlaid image withrespect to the object behind the display screen stays consistent even asthe user moves their head and/or the object behind the display screen.

For the embodiments shown in FIGS. 1-3 and 5-6, what content or image510 is displayed on the display screen is at least in part dependentupon a user viewpoint. Based on the user viewpoint, the display system100 determines where and what parts of the object located behind thescreen appear on the display. When rendering based on a different userviewpoint, different parts/different views of the object are displayed.If the user changes his position (angle from the display screen,distance from the display screen, etc.) then his viewpoint (V) from thescreen changes. If the content displayed on the screen is being renderedfrom a different viewpoint, the overlaid image 510 appears at adifferent point on the screen.

Step 430 shown in FIG. 4A is the step of generating content for displayon a display screen wherein the location of the content is based on theuser viewpoint location and the behind screen object location. Step 430is shown in expanded detail in FIG. 4B. Referring to 4B, step 430 (thestep of generating content for display) further includes the steps ofdetermining the augmented image to be displayed (step 440); determiningthe line of sight between the user viewpoint location and the behindscreen object location (step 450); and for each pixel of the augmentedimage determining along the line of sight, the pixel location of theaugmented image on the plane of the display screen (step 460).

Referring again to FIG. 3 shows a view of the points required togenerate content on the display screen. In one embodiment, the locationdesignated by the symbol P is found by determining the line of sight(the line) between two points—where the two points are (1), the point“O”, the location of the object behind the screen and (2), the point“V”, the user's viewpoint. The display system can then determine thepixel locations (P) on the display screen that correspond to the user'sline of sight between the two points. This allows the generated contenton the display screen to be registered to real objects behind thescreen. Further, the display system recognizes the position of theuser's hands (in this case the object) with respect to the projectedelements (content displayed on the display screen) in the scene.

FIG. 5A shows a side perspective view of a desktop version of anaugmented display system where the user is manipulating a physicalobject behind the display screen according to one embodiment of theinvention. In the embodiment shown in FIG. 5A, there are two physicalobjects located behind the display screen. The first physical object isthe user's hand 120 a. The second physical object is a rectangular block120 b being held or manipulated by the user's hand 120 a. In theembodiment shown in FIG. 5A, the overlaid image 510 is an image of acar.

FIG. 5B shows a front perspective view of the same configuration shownin FIG. 5A, where the view on and through the screen is intended torepresent the view presented to the user. As can be clearly seen fromFIG. 5A, the overlaid image of the car 510 appears on the display screen112—but is not part of the physical object 120 a, 120 b behind thescreen. However, when the user is viewing the object from the front ofthe transparent display screen (the front perspective shown in FIG. 5B),it appears to the user that the car 510 is overlaid within therectangular block 120 b (the physical object.) Thus, the display systemsimulates an object (a block) where content (a car) is overlaid onto thephysical object behind the display screen. As the user changes hisviewpoint with respect to the object (the rectangular block 120 b), theoverlaid object (the car 510) on the display screen 112 moves to followthe physical object position.

In the embodiment shown in FIGS. 5A and 5B, the user is shownmanipulating physical objects (his hand 120 a, a rectangular open cube120 b). In an alternative embodiment shown in FIG. 6A, the displaysystem is used to provide a means for the user to view, interact withand manipulate virtual objects. The virtual objects are shown as imageson the display screen (referred to as an overlaid image or augmentedimage).

Referring to FIG. 6A shows a side perspective view of a desktop versionof the display system with two virtual objects overlaid on the displayscreen. In the embodiment shown, the physical object tracked behind thedisplay screen is the user's hand. In the embodiment shown in FIG. 6A,the user is manipulating at least one virtual object—the uppermost cube610 a.

Referring to FIG. 6B shows a front perspective view of the augmenteddisplay system shown in FIG. 6A, where the view on and through thescreen is intended to represent the view presented to the user. In theembodiment shown in FIG. 6A, the user's hand is shown manipulating acube (a virtual object 510 a). Although the virtual object is actuallyan overlaid image that is located on the surface of the display screen112, in one embodiment the object 510 a is simulated to appear in 3Dbehind the display screen. In one example, the impression of depth(appearance of object behind the display screen) is created simply byusing the motion parallax that occurs naturally as part of the displaygeneration component 126. For instance, if the virtual object 510 a isrendered so as to appear to be behind the user's hand, then—as the usermoves their head from side to side—that object will appear to move lesswith respect to the screen than the user's hand. By tracking the user'shead position, the display screen can convey an impression of depth in ascene using motion parallax.

The embodiment shown in FIGS. 6A and 6B, show a user manipulating avirtual object (a cube) behind the display screen. As shown, portions ofthe user's hand manipulating the virtual object are occluded by thevirtual object. In order to implement occlusions with respect to theoverlaid image shown on the screen, the display system needs detailed 3Dinformation about the physical object positioned behind the screen.Based on the information about the 2D overlaid image (the virtual objector cube) displayed on the screen and the detailed 3D information aboutthe object behind the screen, the occlusion component 186 of the displaygeneration component 126 determines what parts of the overlaid imageshould be blocked by or occluded on the display screen and what parts ofthe overlaid image should be displayed.

In one embodiment, the detailed 3D information about the object behindthe screen (the user's hand) is provided by depth cameras. In otherwords, in this example the object tracking sensors 148 a, 148 b would bedepth cameras. Since the sensors described above can providethree-dimensional information on the position and shape of the user'shand, the display system 100 can determine what parts of the computergenerated content the user's hand should occlude and not display thoseparts on the overlaid image of the cube 610 a. Thus, simulatingocclusion from the point of view of the user.

Also provided to the display system 100 is a description of the overlaidimage 184 that is shown on the display screen. In one embodiment, thedescription of the overlaid image 184 is a 2D description of the objectavailable for user interaction or manipulation. In one embodiment, thedescription of the overlaid image 184 is a 3D description of the objectavailable for user interaction or manipulation. In one example, thedescription of the overlaid image is provided by the display system andanother example, the description of the overlaid image is provided bythe user.

In one example, graphic rendering software 190 capable of rendering a 3Doverlaid object on a 2D screen is included in the display generationcomponent. Thus a 2D description of the overlaid image can be convertedinto a 3D overlaid object on a 2D display screen. Assume an examplewhere the display system has a 3D description of the physical objectbehind the screen and a 3D description of the 2D overlaid imagedisplayed on the display screen. Using the 3D information provided forthe overlaid image, the display system can simulate/determine how theoverlaid virtual image interacts with a physical object positionedbehind the screen. From this simulation, determinations can be madeabout which areas of the virtual image and which areas of the physicalobjects are shown and which areas are occluded, as well as how theuser's movements result in interacting with the virtual content (e.g.has the user grasped a virtual object, allowing them to move it).

As previously stated with respect to the image shown in FIGS. 6A and 6B,part of the virtual cube on the display that the user is manipulating isoccluded by the user's fingers. For the transparent screenimplementation described, the 3D information provided allows the displaysystem to determine which parts of the overlaid image are erased (wouldnot be shown if you were manipulating the object in 3D space.) Objectsthat are not shown are transparent on the transparent display screen.

FIG. 7A shows a perspective front view of a handheld version of anaugmented reality display system according to an embodiment of theinvention. The display screen of the embodiment shown in FIG. 7A may ormay not be transparent. In one embodiment, instead of being atransparent screen, the display screen 112 is a conventional opaquescreen.

In this opaque screen implementation, the display screen attempts tosimulate a transparent screen by capturing the view of the physicalobject 710 a located behind the screen and displaying this content onthe display screen 112. This captured image provides an approximation ofthe actual view that would be shown by a transparent screen, but with anoffset viewpoint. Instead of being the captured image reflecting theuser's view through the transparent screen, the viewpoint seen on thedisplay screen is from the perspective of a camera 148 c positioned onthe back of the display screen.

In the embodiment shown in FIG. 7A, the at least one object trackingsensor is a video camera positioned to capture the scene behind thedisplay screen. The display system simulates a transparent screen byoutputting on the display screen of the handheld device—the image of theobject behind the display screen captured by the object tracking sensor148 c in real time. In the example shown in FIG. 7A, the image displayedon the display screen is identical to a portion of the image captured bythe object tracking sensor 148 c.

For the opaque screen embodiment, at least one object tracking sensor isa video camera. In one embodiment (not shown), a single object trackingsensor is used. In this embodiment, the object tracking sensor is adepth video camera or other image capture device capable of (1)providing information about the object location (O) and (2) providing avideo image of the area or scene behind the display screen. In anotherembodiment (shown in FIG. 7A), the object tracking sensors 148 a and 148b are used to track the object location behind the display screen andthe object tracking sensor 148 c is used to provide video of the scenebehind the display screen.

In one embodiment, the image output to the display (the imagetransmitted to simulate the transparent screen) is modified. Forexample, if the sensor information about the user position in front ofthe display screen is available, the system could try to modify theimage to more closely approximate the viewpoint of the user—instead ofthe viewpoint of the object tracking sensor 148 c.

FIG. 7B shows a flow diagram for a method 700 of generating content fordisplay on the opaque display screen shown in FIG. 7A according to anembodiment of the invention. The method shown in FIG. 7B includes thesteps of: determining a user viewpoint based on data collected by theviewpoint assessment sensors (step 710); determining the location (andoptionally, shape and orientation) of an object based on data collectedfrom an object tracking sensor (step 720); collecting video image dataof the scene behind the display screen (step 730); and generating amodified video image, where the modified video image is modified basedat least on the user viewpoint and the object location (step 740).

Steps 710 and 720 are similar to the steps 410 and 420 in the method400. However, in the implementation described with respect to FIG. 7A,an additional step of collecting a video image is required. Although avideo image may be collected in the method 400, it is not required togenerate content. In the implementation of FIG. 7A since the screen isnot transparent, the video image is used to simulate what would appearto a user through a transparent screen. The video image is modified sothat only the part of the view (from the video image) that the userwould normally see through the transparent screen is displayed. Thevideo image is modified based on the user viewpoint and object location.Typically this is accomplished by modifying the video to take intoaccount the viewpoint of the user, the extent of the scene that isoccluded by and therefore needs to be displayed on the screen, and thelocation of the object behind the screen.

FIG. 8 shows a computer system for implementing the methods shown inFIG. 4 and described in accordance with embodiments of the presentinvention. It should be apparent to those of ordinary skill in the artthat the method 400 represents generalized illustrations and that othersteps may be added or existing steps may be removed, modified orrearranged without departing from the scopes of the method 400. Thedescriptions of the method 400 are made with reference to the system 100illustrated in FIG. 1 and the system 800 illustrated in FIG. 8 and thusrefers to the elements cited therein. It should, however, be understoodthat the method 400 is not limited to the elements set forth in thesystem 800. Instead, it should be understood that the method 400 may bepracticed by a system having a different configuration than that setforth in the system 800.

Some or all of the operations set forth in the method 400 may becontained as utilities, programs or subprograms, in any desired computeraccessible medium. In addition, the method 400 may be embodied bycomputer programs, which may exist in a variety of forms both active andinactive. For example, they may exist as software program(s) comprisedof program instructions in source code, object code, executable code orother formats. Any of the above may be embodied on a computer readablemedium, which include storage devices and signals, in compressed oruncompressed form.

FIG. 8 illustrates a block diagram of a computing apparatus 800configured to implement or execute the methods 400 depicted in FIGS. 4A,4B and 7B, according to an example. In this respect, the computingapparatus 800 may be used as a platform for executing one or more of thefunctions described hereinabove with respect to the display controllercomponent 130.

The computing apparatus 800 includes one or more processor(s) 802 thatmay implement or execute some or all of the steps described in themethods 400. Commands and data from the processor 802 are communicatedover a communication bus 804. The computing apparatus 800 also includesa main memory 806, such as a random access memory (RAM), where theprogram code for the processor 802, may be executed during runtime, anda secondary memory 808. The secondary memory 808 includes, for example,one or more hard drives 810 and/or a removable storage drive 812,representing a removable flash memory card, etc., where a copy of theprogram code for the method 800 may be stored. The removable storagedrive 812 reads from and/or writes to a removable storage unit 814 in awell-known manner.

Exemplary computer readable storage devices that may be used toimplement the present invention include but are not limited toconventional computer system RAM, ROM, EPROM, EEPROM and magnetic oroptical disks or tapes. Concrete examples of the foregoing includedistribution of the programs on a CD ROM or via Internet download. In asense, the Internet itself is a computer readable medium. The same istrue of computer networks in general. It is therefore to be understoodthat any electronic device and/or system capable of executing thefunctions of the above-described embodiments are encompassed by thepresent invention.

Although shown stored on main memory 806, any of the memory componentsdescribed 806, 808, 814 may also store an operating system 830, such asMac OS, MS Windows, Unix, or Linux; network applications 832; and adisplay controller component 130. The operating system 830 may bemultiparticipant, multiprocessing, multitasking, multithreading,real-time and the like. The operating system 830 may also perform basictasks such as recognizing input from input devices, such as a keyboardor a keypad; sending output to the display 820; controlling peripheraldevices, such as disk drives, printers, image capture device; andmanaging traffic on the one or more buses 804. The network applications832 includes various components for establishing and maintaining networkconnections, such as software for implementing communication protocolsincluding TCP/IP, HTTP, Ethernet, USB, and FireWire.

The computing apparatus 800 may also include an input devices 816, suchas a keyboard, a keypad, functional keys, etc., a pointing device, suchas a tracking ball, cursors, etc., and a display(s) 820, such as thescreen display 110 shown for Example in FIGS. 1-3 and 4-7. A displayadaptor 822 may interface with the communication bus 804 and the display820 and may receive display data from the processor 802 and convert thedisplay data into display commands for the display 820.

The processor(s) 802 may communicate over a network, for instance, acellular network, the Internet, LAN, etc., through one or more networkinterfaces 824 such as a Local Area Network LAN, a wireless 802.11x LAN,a 3G mobile WAN or a WiMax WAN. In addition, an interface 826 may beused to receive an image or sequence of images from imaging components828, such as the image capture device.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that the specificdetails are not required in order to practice the invention. Theforegoing descriptions of specific embodiments of the present inventionare presented for purposes of illustration and description. They are notintended to be exhaustive of or to limit the invention to the preciseforms disclosed. Obviously, many modifications and variations arepossible in view of the above teachings. The embodiments are shown anddescribed in order to best explain the principles of the invention andits practical applications, to thereby enable others skilled in the artto best utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the scope of the invention be defined by the followingclaims and their equivalents:

What is claimed is:
 1. A display system (100) the system comprising: adisplay (110), including a display screen (112); a viewpoint assessmentcomponent (116) to determine a viewpoint of a user positioned in frontthe display screen; and an object tracking component (124) to track theuser manipulation fan object positioned behind the display screen. 2.The display system recited in claim 2 further including a displaygeneration component (126), wherein based on data from the viewpointassessment component and the object tracking component, the displaygeneration component creates content for display on the display screen.3. The display system recited in claim 2 wherein the display screen iscapable of operating in a transparent mode.
 4. The display systemrecited in claim 3, wherein the object tracking component is capable ofreceiving data from at least one object tracking sensor, the at leastone object tracking positioned to capture an object behind the displayscreen.
 5. The display system recited in claim 4, further wherein theobject located behind the screen is registered with an augmented imageon the display screen.
 6. The display system recited in claim 2, whereinthe viewpoint assessment component is capable of receiving data from atleast one viewpoint assessment sensor, the at least one viewpointassessment sensor facing outwards from the front of the display screento capture a user positioned in front of the display screen.
 7. Thedisplay system recited in claim 6, wherein the captured data from the atleast one viewpoint assessment sensor is used to determine the user'seye position.
 8. The display system recited in claim 7 wherein based onuser eye position, the viewpoint of the user is determined.
 9. Thedisplay system recited in claim 8 wherein based on the user viewpointand object location, the line of sight is of the user is determined,wherein based on the line of sight the augmented image is displayed inthe plane of the display screen.
 10. The display system recited in claim3 wherein the display screen is capable of operating in an opaque mode.11. The display system recited in claim 10 wherein at least one of theobject tracking sensors is a video camera and wherein the output of thevideo camera is output as content onto the display screen.
 12. Acomputer readable storage medium having computer readable programinstructions stored thereon for causing a computer system to performinstructions, the instructions comprising the steps of: to determine auser viewpoint based on data collected from viewpoint assessment sensors(410); to determine an object location of an object capable of beingmanipulated behind a display screen, the location of the objectdetermined based on data collected from object tracking sensors (420);and to generate content for display, wherein the content location isdetermined based upon user viewpoint and object location (430).
 13. Thecomputer readable storage medium recited in claim 12, wherein thecontent location is determined by determining the line of sight betweenthe user viewpoint and the location of the object positioned behind thedisplay screen.
 14. The computer readable storage medium recited inclaim 13, wherein for each pixel of an augmented image to be displayed,the pixel location of the augmented image is located along the line ofsight in the plane of the display screen.
 15. A computer readablestorage medium having computer readable program instructions storedthereon for causing a computer system to perform instructions, theinstructions comprising: to determine a user viewpoint based on datacollected from viewpoint assessment sensors (710); to determine objectlocation based on data collected from object tracking sensors (720); tocollect a video image of the area behind the display screen (730); andto generate a modified video image where the modified video image isbased on the user viewpoint and object location (740).
 16. The computerreadable storage medium recited in claim 15, wherein the modified videoimage is output on the display screen.